I have no idea how old you are but it is not unusual for parents to have children in their late 30's and even 40's. I would not worry about that unless her doctors are concerned for health reasons.
I would not do the IVF with a history of breast cancer especially if she has the type that is hormone driven, her doctor should have told her if that is the case. Some breast cancer can be driven by up to three hormones and taking those drugs are like giving the cancer vitamins. My doctor told me to stay away from all hormones because of that, I even try to stay away from things that have bpa in them because they can act as hormones. Even though your wife is in remission and they got it all she still needs to be careful because there is always that percentage that it can come back.
Going the route you went through for adoption can bring heartache as you experienced, you can always go through a reputable adoption agency.
I would suggest you meet with your wife's doctors and discuss what would be safe and healthy for her body because of her history. There is also another thing you can look forward to and that is grandchildren! They will bring you so much joy when the time is right, again I am not sure how old your children are.
I hope you find the best solution for your family.